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ABSTRACT 



This brief paper summarizes three papers and a response 
presented at a symposium examining longitudinal comparison studies of 
federally funded community mental health services (CMHS) for children and 
their families. Emphasis was on comparing the system of care approach to a 
more traditional approach. The symposium provided an update on the status of 
the studies by: (1) addressing the policy implications of these studies; (2) 

providing an update on outcomes study data collected; (3) reporting on data 
collected on families' services experiences; and (4) comparing service system 
development between CMHS-funded and comparison communities. The paper 
provides summaries of the following presentations: an introduction by E. 

Wayne Holden, Chair; an overview on the current status of the longitudinal 
comparison studies by Rolando L. Santiago, Director of Program Evaluation in 
the Child, Adolescent and Family Branch at the Center for Mental Health 
Services; a report by Mario Hernandez, University of South Florida, on the 
development and results of the Systems of Care Practice Review, a measure 
implemented in the Phase I comparison studies to evaluate families ' 
experiences with care; and a report by Ana Maria Brannan of ORC Macro on 
preliminary results of systems level assessments conducted in early 1999 
across the six communities participating in the longitudinal comparison 
study. Discussion by Trina W. Osher, a parent and advocate, raises questions 
about the generalization of the findings (which supported the system of care 
approach) and urges greater refinement of evaluation systems. (DB) 
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Symposium Overview 

Preliminary Report on the Longitudinal 
Comparison Study of the National 
Evaluation of the Comprehensive 
Community Mental Health Services for 
Children and Their Families Program 



Chair 

E. Wayne Holden 

Discussant 
Trina W. Osher 

Presenters 
Rolando L. Santiago 
Mario Hernandez 
Ana Maria Brannan 



Introduction 

E. Wayne Holden 

The longitudinal comparison studies, initiated in 1997, ate a 
component of the National Evaluation of the Comprehensive 
Community Mental Health Services for Children and their Families 
Program. The purpose of the comparison study is to address the primary 
question: “Do children with behavior and functioning problems improve 

more when served with a system of care approach C °™P^ Versed in these studies, including what are 
traditional approach?” Several secondary ^ ueSt ‘°™ Center for Mental Health Services (CMHS)-funded 
the differences in service system development betwee , iences differ in systems of 

communities and their matched comparisons, an update on the status of the longitudinal 

care compared to other communities. This symposium P™ W “ iding „ upda ? e on 
comparison studies by, 1) addre* ng rhe policy experiences, "and 4) 

ourcome. study dara collect, on 3) reporting on a^mparison communities. 

comparing service system development between CMHS-tunded and comp 

™ in child™. mental heal* service. and *e need for evaluation da,, to 

sunnort the expansion of community-based services. 
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ngitudinal comparison studies . 0 § P . rnmnar ; SO n studies was briefly summarized, 

t Sevan, litetatute, the current status of *e long,^ c omp^ .J~*« ™ ^ 

ince 1997, over 1,000 iimilie. have been teetutted from ^ ^parld i„ , 

immunities to participate in Phase comparison mu i- j costs data are currently being 

stems level assessment and a family completed in December, 2000, 

ampiled. Follow up outcomes data col ec differences in demographic and clinical 

reliminary analyses of intake data suggest ** * ere , a . outcomes analyses Data completion rates for 
haracteristics at intake that will need ate witlJthe acceptable 

he Phase 1 comparison studies at the 6 and r 1999 i nv0 lving two system of care sites 

Two “7"n f99T»dr m“he'd £££. S— l*i A different 

hat were miually funded in 1 ))/ ana . • i st udv is being implemented that 

mtcomes data collection protocol has been imp eme " ® ^ families for several months after services are 

xamines interactions between service delivery personnel ™ effec[s of evidence based 

uitiated Treatment effectiveness studies that focus on measuring the specihc i ™ 1999 

interventions within systems of care are being planned for grantees hinded in 1998 and 1999. 

Mario Hernandez, University of South ^^22^ A* ^wl^compa^™ sZdies to 
Systems of Care Practice Review, a measure i «*/> of service experiences at the practice level 

evaluate families’ experiences with care. . le$ werg being directly expressed in practice- 

was included to determine whether system P r s ^ 0 f care principles may be fully 

vel interactions among service deliveij personn • incomplete infusion of these principles 

pressed at the overall community and specific agency levels, but incomp 
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into practice interactions will limit tests of the overall effectiveness of the approach. An assessment of 
service experiences at the individual child and family level allows the testing of potential differences 
between system of care and comparison sites and the examination of how service experiences overall may 
influence the relationships between service delivery systems and outcomes. Case studies were used as the 
approach to evaluate service experiences due to their unique applicability for investigating social 
phenomena in real world settings. The specific case study protocol for this study (University of South 
Florida, 1998) provided both qualitative and quantitative information about families’ experiences at the 
interface between services provided and the needs and strengths of the children and families. 

Samples of families were selected from system of care and comparison sites for participation in the 
systems of care practice review protocol. The protocol for each family participating in this part of the 
longitudinal comparison study consisted of multiple data collection components including document 
review, primary caregiver interview, child interview, formal provider interviews and informal helper 
interviews. Interviews and document reviews were conducted by data collection teams who typically 
conducted a 1- to 2-week data collection site visit to obtain information for all participating families. 
Information was combined across the document review and interview data using a sequential analysis 
process that included coding, sorting, rating and examining the information collected by the data 
collection team. The primary domains that were assessed included: 1) child and family, 2) community, 3) 
cultural competency and 4) impact. Preliminary results revealed significant differences between samples 
from systems of care and comparison sites across the domains assessed by the practice review protocol. All 
differences indicated higher levels of system of care principles in CMHS-funded sites. 

Ana Maria Brannan of ORC Macro presented the preliminary results of systems level assessments 
conducted in the Spring of 1999 across the six communities participating in the longitudinal 
comparison study . The conceptual model underlying the systems level assessment protocol and the 
procedures for conducting the visits were reviewed. Quantitative results from site visits conducted in the 
Spring of 1999 were presented along with qualitative data to enrich the understanding of differences 
detected in levels of system development between systems of care and their comparison communities. In 
general, funded systems of care received higher scores on average than systems without funding, 
especially in the infrastructure domain. Across the longitudinal comparison study pairs, interagency 
involvement and community-based service delivery were areas of greatest difference with CMHS-funded 
systems of care displaying higher levels of development in these areas. Family involvement, however, was 
a distinguishing feature in only one of the three longitudinal comparison study pairs. Evaluating the 
relationships between systems characteristics and outcomes awaits the completion of outcomes data 
collection in early 2001. 

Discussants’ comments focused on the strengths and weaknesses of the longitudinal comparison 
study and the implications of these results for the field of children’s mental health services more generally. 

Discussion 

Trina W. Osher, M. A. 

My frame of reference is that of an advocate and parent of children who use mental health services. 

I subscribe to the values and principles of the system of care — though my own family did not have 
the opportunity to benefit from such a system directly. The system of care is not a treatment or 
services, but an structured approach to service design that we assume and hope makes it easier, more 
efficient, and more effective to deliver services for children who have serious emotional disorders and 
their families. Families who experience systems of care tell me that they like them, that they think they 
are better for their children. Personal testimony to the efficacy of a systems of care approach is great, : 
but, in a political arena where I seek support and funding for children’s mental health — and systems 
of care specifically — I need some hard evidence to convince others. !- 

Do these studies help me out? Frankly, I’m not sure! I confess to finding the case study methodology 
of the System of Care Practice Review family friendly and appealing — though it does seem to require a 
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great deal of effort and cost a lot to use. The findings also appeal to me — families in grantee 
communities where there is a highly developed system of care report having had a better experience; and 
in particular that they were included in planning services and that the approach was strengths based. It is 
also good news that families felt their children made greater improvements in the grantee communities. 

But given the limitations of the study I hardly think the findings can be generalized. Furthermore, 
without linking the process of receiving services to outcomes and their cost, I doubt that these findings 
are powerful enough to influence national policymakers. 

The system development comparison shows that there are in fact measurable differences between 
systems of care and traditional service communities — particularly with respect to infrastructure and 
community-based programming. This too is good news. But it is still not sufficient to impact policy 
makers. I confess to being disappointed to learn that family involvement was a distinguishing feature in 
only one comparison pair. Why is this so? How is this grant community different from the others with 
respect to the features that support family participation? I hope that this does not mean that we think we 
can build systems of care without significant family involvement. My understanding is that the system of 
care must have families meaningfully involved in governance and decision making about development, 
implementation, and evaluation of such systems. 

This study raised some further questions about its implications. Do these results about family 
involvement mean that there needs to be a more intensive effort to include families as full decision 
making partners in systems of care? Do grant communities need help recruiting, training, supporting, 
and retraining families on their team? 

Like any ordinary non-research person, 1 look for a message in the medium. In this case, I look for 
pictures or images created by the graphs presented. While the curves for the grantee communities were 
generally above the curves of the comparison communities, the shape of the comparison curves was 
similar for some of the pairs. This suggests to me that when federal grant resources are added to a 
community, they enhance existing services to get more of the same services without actually creating a 
change in the infrastructure, services, or outcomes. My own view of the system of care approach is not 
just that it involves a larger number of the elements commonly found in communities, but that it 
arranges and uses these elements in different ways; systems of care break down barriers and create new 
service delivery systems. Do these findings mean that there is something (structural, political, economic, 
or social) operating in all six of these communities that the system of care approach taken by the grantee 
communities is not powerful enough to overcome? Or does this mean that the instrument or the analysis 
tools are not capable of discerning significantly meaningful differences in infrastructure? What about 
conducting comparisons within the same community? Since grant funding is limited, not all children 
who might be eligible would be enrolled in a system of care. Concerns about the grant influencing other 
community structures and, therefore, contaminating the comparison may not be all that important since, 
as these comparisons show, systems of care take a long time to develop. Indeed, none of the grant 
communities in these comparison studies had fully achieved all dimensions of a system of care. 

How children and their families experience systems of care and how communities build them is 
important to understand and I applaud researchers in the field of childrens mental health for 
contributing to this understanding. But this is still early, pioneering work. Once these measures and 
approaches are refined and used further, the next step will be to relate this work to the changes that occur 
in the lives of the children and families served. Even a true believer in systems of care like myself wants 
evidence not only that such systems exist, but that they make a significant difference. I sincerely hope 
that the research community succeeds in finding the significant and distinguishing features of systems of 
care and further finds that these result in better outcomes that are cost effective and satisfying for 
children, families, providers, and communities. 
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